Data beam/batch, dim tags per batch, RecLayer time better by albertz · Pull Request #613 · rwth-i6/returnn

albertz · 2021-09-02T07:46:36Z

No description provided.

albertz · 2021-09-02T09:05:03Z

Many tests are failing. (FAILED (errors=18, failures=15) in test_TFEngine).

albertz · 2021-09-02T09:05:07Z

    line: time_dim_tag.set_tag_on_size_tensor(fixed_seq_len, batch=output_template.output.batch)
    locals:
      time_dim_tag = <local> DimensionTag{'time:var:extern_data:data'[B]}
      time_dim_tag.set_tag_on_size_tensor = <local> <bound method DimensionTag.set_tag_on_size_tensor of DimensionTag{'time:var:extern_data:data'[B]}>
      fixed_seq_len = <local> <tf.Tensor 'output/rec/subnet_base/check_seq_len_batch_size/check_input_dim/identity_with_dim_check:0' shape=(?,) dtype=int32>
      batch = <not found>
      output_template = <local> <_TemplateLayer(LinearLayer)(:template:linear) output/'output' out_type=Data(batch_shape_meta=[B?,F|F'feature:output_output'(2)]) (construction stack None)>
      output_template.output = <local> Data(name='output_output', batch_shape_meta=[B?,F|F'feature:output_output'(2)])
      output_template.output.batch = <local> None
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 289, in DimensionTag.set_tag_on_size_tensor
    line: assert self.batch and batch
    locals:
      self = <local> DimensionTag{'time:var:extern_data:data'[B]}
      self.batch = <local> BatchInfo{B}
      batch = <local> None
AssertionError:

albertz · 2021-09-02T09:05:57Z

  File "/home/runner/work/returnn/returnn/returnn/tf/network.py", line 1059, in TFNetwork.add_layer
    line: layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)
    locals:
      layer = <not found>
      self = <local> <TFNetwork 'root(extra.search)' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool> search>
      self._create_layer = <local> <bound method TFNetwork._create_layer of <TFNetwork 'root(extra.search)' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool> search>>
      name = <local> 'output', len = 6
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'back_prop': False, 'target': ['classes0', 't_base'], 'size_target': 't_base', 'include_eos': True, 'max_seq_len': <tf.Tensor 'max_seq_len_encoder:0' shape=() dtype=int32>, '_network': <TFNetwork 'root(extra.search)' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool> search>, '_name': ..., len = 12
  File "/home/runner/work/returnn/returnn/returnn/tf/network.py", line 951, in TFNetwork._create_layer
    line: layer_desc["output"] = layer_class.get_out_data_from_opts(**layer_desc)
    locals:
      layer_desc = <local> {'back_prop': False, 'target': ['classes0', 't_base'], 'size_target': 't_base', 'include_eos': True, 'max_seq_len': <tf.Tensor 'max_seq_len_encoder:0' shape=() dtype=int32>, '_network': <TFNetwork 'root(extra.search)' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool> search>, '_name': ..., len = 14
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_class.get_out_data_from_opts = <local> <bound method RecLayer.get_out_data_from_opts of <class 'returnn.tf.layers.rec.RecLayer'>>
  File "/home/runner/work/returnn/returnn/returnn/tf/layers/rec.py", line 415, in RecLayer.get_out_data_from_opts
    line: out = out.copy_template_replace_dim_tag(axis=out.time_dim_axis, new_dim_tag=_time_dim_tag)
    locals:
      out = <local> Data(name='output_output', dtype='int32', sparse=True, dim=8, batch_shape_meta=[T|'unknown_time'[?],B&Beam{'extra.search:output/t0'}(3)])
      out.copy_template_replace_dim_tag = <local> <bound method Data.copy_template_replace_dim_tag of Data(name='output_output', dtype='int32', sparse=True, dim=8, batch_shape_meta=[T|'unknown_time'[?],B&Beam{'extra.search:output/t0'}(3)])>
      axis = <not found>
      out.time_dim_axis = <local> 0
      new_dim_tag = <not found>
      _time_dim_tag = <local> DimensionTag{'time-with-postfix:0_t_target'[B]}
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 2324, in Data.copy_template_replace_dim_tag
    line: return Data(**opts)
    locals:
      Data = <global> <class 'returnn.tf.util.data.Data'>
      opts = <local> {'name': 'output_output', 'dim_tags': (DimensionTag{'time-with-postfix:0_t_target'[B]}, DimensionTag{B}), 'dtype': 'int32', 'time_dim_axis': 0, 'sparse': True, 'dim': 8, 'batch': BatchInfo{B, Beam{'extra.search:output/t0'}(3)}, 'beam': SearchBeam(name='extra.search:output/t0', beam_size=3, depend..., len = 8
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 1372, in Data.__init__
    line: self._adapt_batch_consistent_dim_tags()
    locals:
      self = <local> Data(name='output_output', dtype='int32', sparse=True, dim=8, batch_shape_meta=[T|'time-with-postfix:0_t_target'[B],B&Beam{'extra.search:output/t0'}(3)])
      self._adapt_batch_consistent_dim_tags = <local> <bound method Data._adapt_batch_consistent_dim_tags of Data(name='output_output', dtype='int32', sparse=True, dim=8, batch_shape_meta=[T|'time-with-postfix:0_t_target'[B],B&Beam{'extra.search:output/t0'}(3)])>
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 1643, in Data._adapt_batch_consistent_dim_tags
    line: self._dim_tags = tuple(tag.get_for_batch(self.batch) for tag in self._dim_tags)
    locals:
      self = <local> Data(name='output_output', dtype='int32', sparse=True, dim=8, batch_shape_meta=[T|'time-with-postfix:0_t_target'[B],B&Beam{'extra.search:output/t0'}(3)])
      self._dim_tags = <local> (DimensionTag{'time-with-postfix:0_t_target'[B]}, DimensionTag{B})
      tuple = <builtin> <class 'tuple'>
      tag = <not found>
      tag.get_for_batch = <not found>
      self.batch = <local> BatchInfo{B, Beam{'extra.search:output/t0'}(3)}
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 1643, in <genexpr>
    line: self._dim_tags = tuple(tag.get_for_batch(self.batch) for tag in self._dim_tags)
    locals:
      self = <local> Data(name='output_output', dtype='int32', sparse=True, dim=8, batch_shape_meta=[T|'time-with-postfix:0_t_target'[B],B&Beam{'extra.search:output/t0'}(3)])
      self._dim_tags = <local> (DimensionTag{'time-with-postfix:0_t_target'[B]}, DimensionTag{B})
      tuple = <builtin> <class 'tuple'>
      tag = <local> DimensionTag{'time-with-postfix:0_t_target'[B]}
      tag.get_for_batch = <local> <bound method DimensionTag.get_for_batch of DimensionTag{'time-with-postfix:0_t_target'[B]}>
      self.batch = <local> BatchInfo{B, Beam{'extra.search:output/t0'}(3)}
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 159, in DimensionTag.get_for_batch
    line: assert beam_expanded_base_data
    locals:
      beam_expanded_base_data = <local> None
AssertionError:

albertz · 2021-09-02T09:08:00Z

  File "/home/runner/work/returnn/returnn/returnn/tf/layers/rec.py", line 1213, in _SubnetworkRecCell._construct_template.<locals>.GetLayer.add_templated_layer
    line: layer_desc["output"] = layer_class.get_out_data_from_opts(**layer_desc)
    locals:
      layer_desc = <local> {'n_out': 1, 'activation': None, 'out_type': {'same_dim_tags_as': {'T': DimensionTag{'enc time'[B]}}}, '_network': <TFNetwork 'root/output(rec-subnet)' parent_net=<TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype..., len = 8
      layer_class = <local> <class 'returnn.tf.layers.basic.LinearLayer'>
      layer_class.get_out_data_from_opts = <local> <bound method LayerBase.get_out_data_from_opts of <class 'returnn.tf.layers.basic.LinearLayer'>>
  File "/home/runner/work/returnn/returnn/returnn/tf/layers/base.py", line 235, in LayerBase.get_out_data_from_opts
    line: return cls._base_get_out_data_from_opts(**kwargs)
    locals:
      cls = <local> <class 'returnn.tf.layers.basic.LinearLayer'>
      cls._base_get_out_data_from_opts = <local> <bound method LayerBase._base_get_out_data_from_opts of <class 'returnn.tf.layers.basic.LinearLayer'>>
      kwargs = <local> {'n_out': 1, 'activation': None, 'out_type': {'same_dim_tags_as': {'T': DimensionTag{'enc time'[B]}}}, '_network': <TFNetwork 'root/output(rec-subnet)' parent_net=<TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype..., len = 8
  File "/home/runner/work/returnn/returnn/returnn/tf/layers/base.py", line 331, in LayerBase._base_get_out_data_from_opts
    line: output = Data(**out_type)
    locals:
      output = <not found>
      Data = <global> <class 'returnn.tf.util.data.Data'>
      out_type = <local> {'same_dim_tags_as': {'T': DimensionTag{'enc time'[B]}}, 'name': 'att_energy_output', 'dim': 1, 'dtype': 'float32', 'batch_dim_axis': 0, 'time_dim_axis': 1, 'shape': (None, 1), 'beam': None}, len = 8
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 1373, in Data.__init__
    line: self.sanity_check(assume_complete=False)
    locals:
      self = <local> Data(name='att_energy_output', batch_shape_meta=[B?,T|'enc time'[B],F|F'feature:att_energy_output'(1)])
      self.sanity_check = <local> <bound method Data.sanity_check of Data(name='att_energy_output', batch_shape_meta=[B?,T|'enc time'[B],F|F'feature:att_energy_output'(1)])>
      assume_complete = <not found>
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 1443, in Data.sanity_check
    line: assert tag.batch == tag.dyn_size_ext.batch
    locals:
      tag = <local> DimensionTag{'enc time'[B]}
      tag.batch = <local> None
      tag.dyn_size_ext = <local> Data(name='time:var:extern_data:data:dyn_size', dtype='int32', batch_shape_meta=[B])
      tag.dyn_size_ext.batch = <local> BatchInfo{B}
AssertionError:

albertz · 2021-09-02T10:38:59Z

Rebased. Now even more tests are failing.
test_TFNetworkRecLayer: FAILED (SKIP=8, errors=18, failures=32)

albertz · 2021-09-02T10:40:21Z

Still:

    line: output = cell.get_output()
    locals:
      output = <not found>
      cell = <local> <_SubnetworkRecCell '<network via test_PositionalEncodingLayer_offset_in_rec>/output(rec-subnet)'>
      cell.get_output = <local> <bound method _SubnetworkRecCell.get_output of <_SubnetworkRecCell '<network via test_PositionalEncodingLayer_offset_in_rec>/output(rec-subnet)'>>
  File "/home/runner/work/returnn/returnn/returnn/tf/layers/rec.py", line 1965, in _SubnetworkRecCell.get_output
    line: time_dim_tag.set_tag_on_size_tensor(fixed_seq_len, batch=output_template.output.batch)
    locals:
      time_dim_tag = <local> DimensionTag{'time:var:extern_data:data'[B]}
      time_dim_tag.set_tag_on_size_tensor = <local> <bound method DimensionTag.set_tag_on_size_tensor of DimensionTag{'time:var:extern_data:data'[B]}>
      fixed_seq_len = <local> <tf.Tensor 'output/rec/subnet_base/check_seq_len_batch_size/check_input_dim/identity_with_dim_check:0' shape=(?,) dtype=int32>
      batch = <not found>
      output_template = <local> <_TemplateLayer(SoftmaxLayer)(:template:softmax) output/'output' out_type=Data(batch_shape_meta=[B?,F|F'feature:output_output'(5)]) (construction stack None)>
      output_template.output = <local> Data(name='output_output', batch_shape_meta=[B?,F|F'feature:output_output'(5)])
      output_template.output.batch = <local> None
  File "/home/runner/work/returnn/returnn/returnn/tf/util/data.py", line 289, in DimensionTag.set_tag_on_size_tensor
    line: assert self.batch and batch
    locals:
      self = <local> DimensionTag{'time:var:extern_data:data'[B]}
      self.batch = <local> BatchInfo{B}
      batch = <local> None
AssertionError:

Edit This is part of the rec time dim tag logic. So maybe this needs all the other rec subnet fixes which come later... (in #605)

albertz · 2021-09-02T20:47:57Z

Still:
    line: output = cell.get_output()
...
    line: assert self.batch and batch
    locals:
      self = <local> DimensionTag{'time:var:extern_data:data'[B]}
      self.batch = <local> BatchInfo{B}
      batch = <local> None
AssertionError: 
This is part of the rec time dim tag logic. So maybe this needs all the other rec subnet fixes which come later... (in #605)

But the rec logic on the other side depends on this here (e.g. get_for_batch). So this implies that they must come together.

This was referenced Sep 2, 2021

Data.get_runtime_sanity_check_op extend #610

Merged

Layer base get out data, use dim tags and correct batch #609

Closed

albertz force-pushed the albert-gen-self-att-wip-3 branch from f1c1859 to a44260d Compare September 2, 2021 10:09

albertz mentioned this pull request Sep 2, 2021

DecideLayer fix out dim tag #617

Closed

albertz added 9 commits September 2, 2021 21:34

Data beam/batch encapsulation, adapt dyn sizes, dim tags per batch

4b2307e

Data small static batch dim tag fixes

6a84085

DimensionTag same_as extended logic, same size, better batch logic

4927c61

Data set_dynamic_size adopt potential new dim tag

8abafe4

DimensionTag set_Tag_on_size_tensor reset, extended exception text

d5c6ca8

Data.set_dynamic_size fix and cleanup outdated size beam tiling logic

f04a94c

DimensionTag.set_tag_on_size_tensor exception less verbose graph

438253e

DimensionTag for batch beam tiling fix multiple beams

fd43640

DimensionTag get_for_batch size identity fix control flow ctx

bd83de7

albertz force-pushed the albert-gen-self-att-wip-3 branch from a44260d to bd83de7 Compare September 2, 2021 19:34

albertz and others added 10 commits September 2, 2021 22:22

Data expand beam set base data attrib on tile

68cab2c

TF util new_seq_len cleanup and fixes for better dim tag batch logic

34d44fe

Layer base get out data, use dim tags and correct batch, beam

e768327

UnflattenNdLayer size dim tag, use same_as_before

0389cba

CondLayer size dim tag, use same_as_before

aaea739

DecideLayer fix out dim tag

0afe8b9

DotLayer, fixes for dim tags, batch

47296fe

select search sources, skip beam expanded tensors

178320a

SelectSearchSourcesLayer, fix transform tag batch

fe85269

Rec layer out data, cleanup and fix, no _post_init_output

8c28370

RecLayer target for time-dim-tag only when not search

2ee9e41

albertz added 11 commits September 2, 2021 22:51

Rec subnet time-dim-tag test consistency to template

77924e6

Rec subnet, opt search resolve, fix seq-len without end

3447102

Rec subnet, output, fix seq len tile transpose for beam expand

33c9075

Rec subnet, output, fixed seq len, tag use same_as_before after check

446bc34

Rec subnet, output, fix time dim tag with dyn end

20b9a6c

Rec subnet output sizes loop acc layer, fix beam tiling, cleanup

099b44b

Rec subnet, cleanup and fix outdated size_placeholder fixup code

bec61fc

Rec subnet, better check for same time

f4f7005

Rec subnet, output layers moved out, fix time dim tag

0d80173

Rec subnet, use correct resolved time dim

2b91629

Rec subnet resolve directly set _RETURNN_dyn_size_beam

1e447a9

albertz changed the title ~~Data beam/batch encapsulation, adapt dyn sizes, dim tags per batch~~ Data beam/batch, dim tags per batch, RecLayer time better Sep 2, 2021

albertz merged commit 523c39c into master Sep 2, 2021

albertz deleted the albert-gen-self-att-wip-3 branch September 2, 2021 21:19

This was referenced Sep 2, 2021

(Extend some tests) (Earlier this was a huge PR) #605

Merged

CumConcatLayer #589

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data beam/batch, dim tags per batch, RecLayer time better#613

Data beam/batch, dim tags per batch, RecLayer time better#613
albertz merged 31 commits intomasterfrom
albert-gen-self-att-wip-3

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021 •

edited

Loading

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

albertz commented Sep 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

albertz commented Sep 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

albertz commented Sep 2, 2021 •

edited

Loading